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DETAILED ACTION 
Response to Arguments 

1 . Applicant's arguments witli respect to claims 1 , 10, 18, 24, 37, and 46 have been 
considered but are moot in view of the new ground(s) of rejection. 

Regarding claims 1, 37, and 46, applicant argues on page 12 that the cited references 
do not disclose or render obvious transferring the information from a first remote device 
to a second remote device; the examiner agrees. Ayatsuka teaches a user may 
remotely control a device by viewing it though a camera. In one example, a user may 
"hyper-drag" a representation of a transparency/slide either from a computer to a 
projector or vice versa (i.e. from the projector to the computer) as long as the computer 
has an image of the projector. Ayatsuka also teaches that this exchange of information 
may happen between a computer and 2 projectors if the computer is linked to both 
projectors via a gaze-link (i.e. a camera view). Therefore, a user may drag a 
representation of a transparency/slide from projector A to projector B because the two 
are connected through a 2-way (i.e. information can be sent or retrieved) path between 
the computer and each projector (transferring the information from the first remote 
physical device to the second remote physical device). Moreover, the exchange of 
information is preceded by a determination as to whether the target device can receive 
transparencies (querying the second remote physical device to determine if it can 
receive the information from the first remote physical device). 
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Regarding claims 10, 18 and 24, applicant argues on page 14 that the references of 
record do not disclose an annotation being displayed on a physical device at the 
location shown in the live video image, such that the annotation can be viewed at the 
location. The examiner agrees; However, US Patent Application Publication 
2004/0236830 to Nelson et al teaches participants of a conference may use annotation 
software installed on an electronic device to annotate a live view of a physical device 
(figure 8 display screen 216). The annotation will appear to other conference 
participants and will be projected onto display 216 (pg. 8 sect. 0067 and pg.9 sect 
0068). It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the remote annotation capabilities disclosed in Nelson in 
the videoconferencing system taught by Ludwig for the benefit of allowing all 
participants of a conference to input and share all of the resources used to 
communicate information in the conference (such as a whiteboard). 

Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or deschbed as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

6. Claims 1-5, and 9 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Ludwig et al (US 7,185,054 B1) in viewof Ayatsuka (US 7,188,139 B1). 
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Regarding claim 1, Ludwig teaches a method for exchanging information 
in a shared interactive environment (abstract) comprising: 
a first live video image and a second live video image (figure 40 element 203 
shows multiple live video images) 

Ludwig does not teach selecting a first remote physical device In a first live 
video Image wherein the first remote physical device has information associated 
with it; 

causing the information to be transferred to a second physical device in a second 
live video Image wherein the transfer is brought about by manipulating a visual 
representation of the information by interacting with the first live video Image and 
the second live video image, 

querying the second remote physical device to determine if it can receive the 
Information from the first remote physical device, and 
transferring the information from the first remote physical device to the second 
remote physical device; 

wherein at least one of the first remote physical device and the second physical 
device has a statically or dynamically defined hotspot in the first live video image 
and or the second live video Image; 

wherein the first remote physical device and the second remote physical device 
are part of the shared interactive environment; and 

wherein the first remote physical device and the second remote physical device 
are different remote physical devices. 
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Ayatsuka teaches a method of remotely controlling devices that includes 
selecting a first remote (col. 14 II. 43-47) physical device (fig. 15 projector 50A) in 
a first live video image wherein the first remote physical device has information 
associated with it (fig. 15 slide A); 

causing the information to be transferred to a second physical device (fig. 15 
projector 50B) in a second live video image wherein the transfer is brought about 
by manipulating a visual representation of the information by interacting with the 
first live video image and the second live video image (hyper-drag as defined in 
fig. 2, col. 21 II. 7-17 and fig. 16. A user may drag a representation of a 
transparency/slide from projector A to projector B because the two are connected 
through a 2-way (i.e. information can be sent or retrieved) path between the 
computer and each projector), 

querying the second remote physical device to determine if it can receive the 
information from the first remote physical device (col. 21 II. 1-4), and 
transferring the information from the first remote physical device to the second 
remote physical device (col. 21 II. 7-11); 

wherein at least one of the first remote physical device and the second physical 
device has a statically or dynamically defined hotspot in the first live video image 
and or the second live video image (col. 21 II. 11-14, see fig. 2 for implementation 
of hyperdrag requiring a representation of a digital object/hotspot); 
wherein the first remote physical device and the second remote physical device 
are part of the shared interactive environment (both of the projectors are part of 
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the network of linked devices and are part of a live video view); and 
wherein the first remote physical device and the second remote physical device 
are different remote physical devices (projectors A and B are two separate 
devices). It would have been obvious to one of ordinary skill in the art at the time 
the invention was made to Include the remote control capabilities taught by 
Ayatsuka In the shared Interactive environment taught by Ludwig for the benefit 
of allowing users to operate various devices in an easy, intuitive, and integrated 
way (col. 6 II. 24-28). 

Regarding claim 2, when read in light of claim 1 , Ayatsuka further teaches 
the manipulation is accomplished by dragging the representation from the first 
physical device and dropping the representation on the second physical device 
(see definition of hyperdrag col. 4 II. 52-62). 

Regarding claim 3, when read in light of claim 1 , limitation "changes to the 
physical devices are visible to all participants In the shared Interactive 
environment" Is Inherent. If all participants in the videoconference have the 
capabilities disclosed in Ludwig in view of Ayatsuka, all participants would have 
the ability to focus on any physical device and obtain current associated 
information, which would reflect any changes. 
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Regarding claim 4, when read in light of claim 1 , Ayatsuka further teaches 
a physical device can include a display (fig. 17), a projector (fig. 15). In regards 
to limitation "a physical device can include a printer, a facsimile machine, a 
personal digital assistant, a computer, and a portable computer", Ayatsuka 
further discloses in 10 II. 33-38 that a computer may link to and control any object 
it sees with a network address (col. 7 II. 25). It is well known in the art that 
printers, facsimile machines, personal digital assistants, computers, and portable 
computers can have IP addresses associated with them. 

Regarding claim 5, when read in light of claim 1 , Ludwig further teaches 
annotating at least one of the first live video image and the second live video 
image (fig 2b, col. 6 II. 57-61 . Updates/annotations are made to the image in real 
time). 

Regarding claim 9, when read in light of claim 1 , Ayatsuka further teaches 
at least one of the first physical device and the second physical device has 
associated with it a pop-up control panel through which a user can configure and 
control it (col. 1011.49-64). 

2. Claims 10-13, 16-17, and 37-46 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ludwig et al (US 7,185,054 B1) in view of Ayatsuka (US 7,188,139 
B1 ), and further in view of Nelson et al (US 2004/0236830). 
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Regarding claim 10, Ludwig teaclies a metliod for exclianging information 
in a sliared interactive environment (abstract), comprising: 
a first live video Image and a second live video image (figure 40 element 203 
shows multiple live video images). 

Ludwig does not teach selecting a first object wherein the first object is 
one of: 1) a physical device in a first live video image and 2) an icon on a 
computing device; 

causing information associated with the first object to be transferred to a second 
object wherein the second object is the other of 1 ) the physical device in a first 
live video image; and 2) the icon on the computing device; 

annotating the first live image; 

automatically transferring the annotation to the physical device if the annotation 
is at least partially drawn over the physical device as it appears in a live video 
image; 

displaying the annotation on the physical device such that the annotation can be 
viewed at the remote location; 

wherein the first physical device has a dynamically defined hotpot in the first live 
video; 

wherein the transfer is brought about by manipulating a visual representation of 
the information; 
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wherein the manipulation includes interacting with the first object and the second 
object; and 

wherein the physical device is part of the shared interactive environment. 

Ayatsuka teaches a method of remotely controlling devices that includes 
selecting a first object wherein the first object is one of 1 ) a physical device at a 
remote location shown in a first live video image, and 2) an icon on a computing 
device (fig. 15 slide. Although the slide icon is not shown, it is understood from 
the discussion in col. 21 II. 7-17 and col. 4 II. 52-62) in a first live video image 
wherein the first remote physical device has information associated with it (fig. 15 
slide A); 

causing information associated with the first object to be transferred to a second 
object wherein the second object is the other one of 1) the physical device at the 
remote location shown in a first live video image (fig. 15 projector 50B) and 2) the 
icon on the computing device; 

wherein at the first physical device has a dynamically defined hotspot in the first 
live video image (col. 21 II. 11-14, see fig. 2 for implementation of hyperdrag 
requiring a representation of a digital object/hotspot); 

wherein the transfer is brought about by manipulating a visual representation of 
the information in the first live video image by interacting with the first live 
video image and the second live video image and the manipulation includes 
interacting with the first object in the first live video image and the second object 
(hyper-drag as defined in fig. 2, col. 21 II. 7-17 and fig. 16. A user may drag a 
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representation of a transparency/slide from the computer to projector B because 
the two are connected through a 2-way path); 

wherein the first remote physical device and the second remote physical device 
are part of the shared interactive environment (both of the projectors are part of 
the network of linked devices and are part of a live video view). It would have 
been obvious to one of ordinary skill in the art at the time the invention was made 
to include the remote control capabilities taught by Ayatsuka in the shared 
interactive environment taught by Ludwig for the benefit of allowing users to 
operate various devices in an easy, intuitive, and integrated way (col. 6 II. 24-28). 

Regarding limitations "annotating the first live image; automatically 
transferring the annotation to the physical device if the annotation is at least 
partially drawn over the physical device as it appears in a live video image; 
displaying the annotation on the physical device such that the annotation can be 
viewed at the remote location", Nelson et al teaches participants of a conference 
may use annotation software installed on an electronic device to annotate a live 
view of a physical device (figure 8 display screen 216). The annotation will 
appear to other conference participants and will be projected onto display 216 
(pg. 8 sect. 0067 and pg.9 sect 0068). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to include the remote 
annotation capabilities disclosed in Nelson in the videoconferencing system 
taught by Ludwig for the benefit of allowing all participants of a conference to 
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input and share all of the resources used to communicate information in the 
conference (such as a whiteboard). 

Regarding claim 1 1 , when read in light of claim 10, Ayatsuka further 
teaches the manipulation is accomplished by dragging the representation from 
the first object and dropping the representation on the second object (see 
definition of hyperdrag col. 4 II. 52-62). 

Regarding claim 12, when read in light of claim 10, limitation "changes to 
the physical device are visible to all participants in the shared interactive 
environment" is inherent. If all participants in the videoconference have the 
capabilities disclosed in Ludwig in view of Ayatsuka and Nelson, all participants 
would have the ability to focus on any physical device and obtain current 
associated information, which would reflect any changes. 

Regarding claim 13, when read in light of claim 10, Ayatsuka further 
teaches a physical device can include a display (fig. 17), a projector (fig. 15). In 
regards to limitation "a physical device can include a printer, a facsimile machine, 
a personal digital assistant, a computer, and a portable computer", Ayatsuka 
further discloses in 10 II. 33-38 that a computer may link to and control any object 
it sees with a network address (col. 7 II. 25). It is well known in the art that 
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printers, facsimile machines, personal digital assistants, computers, and portable 
computers can have IP addresses associated with them. 

Regarding claim 16, when read in light of claim 10, Ayatsuka further 
teaches the information can include a digital file (col. 9 II. 21-24), a sound (col. 9 
II. 36-39), and an audio/video presentation (col. 9 II. 40-46). 

Regarding claim 17, when read in light of claim 10, Ayatsuka further 
teaches the physical device has associated with it a pop-up control panel through 
which a user can configure and control it (col. 10 II. 49-64). 

Regarding claim 37, Ludwig, Ayatsuka, and Nelson all implement their 
inventions in hardware, which inherently requires machine readable instructions, 
therefore a computer readable memory embodying the method described in 
claim 10 is taught by Ludwig in view of Ayatsuka and Nelson (see analysis of 
claim 10). 

Regarding claim 38, when read in light of claim 37, Ludwig, Ayatsuka, and 
Nelson all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer readable memory 
embodying the method described in claim 1 1 is taught by Ludwig in view of 
Ayatsuka, and Nelson (see analysis of claim 11). 
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Regarding claim 39, when read in liglit of claim 37, Ayatsuka, and Nelson 
all implement their inventions in hardware, which inherently requires machine 
readable instructions, therefore a computer data signal embodying the method 
described In claim 12 is taught by Ludwig in view of Ayatsuka, and Nelson (see 
analysis of claim 12). 

Regarding claim 40, when read in light of claim 37, Ayatsuka, and Nelson 
all Implement their Inventions in hardware, which inherently requires machine 
readable instructions, therefore a computer data signal embodying the method 
described in claim 13 is taught by Ludwig in view of Ayatsuka, and Nelson (see 
analysis of claim 13). 

Regarding claim 41, when read in light of claim 37, Ludwig, Ayatsuka, and 
Nelson all Implement their Inventions in hardware, which Inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 14 is taught by Ludwig in view of Ayatsuka, and 
Nelson (see analysis of claim 14). 

Regarding claim 42, when read in light of claim 37, Ludwig, Ayatsuka, and 
Nelson all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
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method described in claim 15 is taught by Ludwig in view of Ayatsuka, and 
Nelson (see analysis of claim 15). 

Regarding claim 43, when read in light of claim 37, Ludwig, Ayatsuka, and 
Nelson all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 10 with the additional limitation of claim 7 is taught by 
Ludwig in view of Ayatsuka, and Nelson (see analysis of claims 10 and 7). 

Regarding claim 44, when read in light of claim 37, Ludwig, Ayatsuka, and 
Nelson all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 16 is taught by Ludwig in view of Ayatsuka, and 
Nelson (see analysis of claim 16). 

Regarding claim 45, when read in light of claim 37, Ludwig, Ayatsuka, and 
Nelson and all implement their inventions in hardware, which inherently requires 
machine readable instructions, therefore a computer data signal embodying the 
method described in claim 17 is taught by Ludwig in view of Ayatsuka, and 
Nelson (see analysis of claim 17). 
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Regarding claim 46 Ludwig, Ayatsuka, and Nelson all implement their 
inventions in hardware, and therefore teach a system with means for carrying out 
the method described in claim 1 is taught by Ludwig in view of Ayatsuka, and 
Nelson (see analysis of claim 1). 

7. Claims 6 and 8 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Ludwig et a! (US 7,185,054 B1) in view of Ayatsuka (US 7,188,139 B1), and further in 
view of Nelson et al (US 2004/0236830). 



Regarding claim 6, when read in light of claim 5, Ludwig in view of 
Ayatsuka teaches the method of claim 5. 

Ludwig in view of Ayatsuka does not teach automatically transferring the 
annotation to a physical device if the annotation is at least partially drawn over 
the physical device as it appears in a live video image. 

Nelson et al teaches participants of a conference may use annotation 
software installed on an electronic device to annotate a live view of a physical 
device (figure 8 display screen 216). The annotation will appear to other 
conference participants and will be projected onto display 216 (pg. 8 sect. 0067 
and pg.9 sect 0068). It would have been obvious to one of ordinary skill in the art 
at the time the invention was made to include the remote annotation capabilities 
disclosed in Nelson in the videoconferencing system taught by Ludwig for the 
benefit of allowing all participants of a conference to input and share all of the 
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resources used to communicate information in tlie conference (such as a 
whiteboard). 

Regarding claim 8, Ludwig in view of Ayatsul<a further teaches the 
information can include a digital file (col. 9 II. 21-24), a sound (col. 9 II. 36-39), 
and an audio/video presentation (col. 9 II. 40-46). 

8. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over Ludwig et 
a! (US 7,185,054 B1) in view of Ayatsuka (US 7,188,139 B1), and further in view of 
Emens et al (US 6,463,343 B1). 

Regarding claim 7, when read in light of claim 1 , Ludwig in view of 
Ayatsuka teaches the method of claim 1 . 

Ludwig in view of Ayatsuka does not teach that the first live video image 
and the second live video image are the same. 

Limitation "the first live video image and the second live video image are 
the same", in light of claim 1 limitation "selecting a first physical device in a first 
live video image 

causing the information to be transferred to a second physical device in a second 
live video image wherein the transfer is brought about by manipulating a visual 
representation of the information" is taught by Emens column 1 lines 57-60 and 
figure 2c, in which a user can select a device to control amongst a plurality of 
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devices in a single live image. It would have been obvious to one of ordinary skill 
in the art at the time the invention was made to incorporate Emen's teaching of 
being able to select a device to remotely control from a plurality of devices in a 
single live image with the invention taught by Ludwig in view of Ayatsuka. This 
combination of inventive ideas would enable videoconference participants to 
remotely control devices in a room without adjusting the camera angle to zoom in 
on each device. 

9. Claims 18 and 21 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Ludwig et al (US 7,185,054 B1) in view of Nelson et al (US 2004/0236830). 

Regarding claim 18, Ludwig teaches a method for annotating a live video 
image wherein the annotation is visible to at least one participant in a shared 
interactive environment (fig 2b, col. 6 II. 57-61 . Updates/annotations are made 
to the image in real time). 

Ludwig does not teach annotating the live video image, wherein the live 
video image shows a location including one or more physical devices at the 
location; automatically transferring the annotation to one of the physical devices 
at the location shown in the live video image if the annotation is at least partially 
drawn over the physical device as it appears in a live video image; and displaying 
the annotation on the physical device such that the annotation can be viewed at 
the location. 
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Nelson et al teaches participants of a conference may use annotation 
software installed on an electronic device to annotate a live view of a physical 
device (figure 8 display screen 216). The annotation will appear to other 
conference participants and will be projected onto display 216 (pg. 8 sect. 0067 
and pg.9 sect 0068). It would have been obvious to one of ordinary skill in the art 
at the time the invention was made to include the remote annotation capabilities 
disclosed in Nelson in the videoconferencing system taught by Ludwig for the 
benefit of allowing all participants of a conference to input and share all of the 
resources used to communicate information in the conference (such as a 
whiteboard). 

Regarding claim 21 , when read in light of claim 18, Ludwig in view of 
Nelson teaches the method of claim 18. 

Ludwig in view of Nelson does not teach the live video image is one of: a 
panoramic view and a zoomed view. 

Official notice is taken that panning and zooming are well known and 
widely used tools In video surveillance and conferencing systems. It would have 
been obvious to one of ordinary skill in the art at the time the invention was made 
to include panning and zooming capabilities in the videoconferencing system 
taught by Ludwig in view of Nelson for the benefit of allowing videoconference 
participants to have access to view of any resources in the collective rooms, such 
as whiteboards or illustrative models. 
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10. Claims 19-20 and 22-23 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ludwig et al (US 7,185,054 B1) in view of Nelson et al (US 
2004/0236830), and further in view of Ayatsuka (US 7,188,139 B1). 

Regarding claim 19, when read in light of claim 18, Ludwig in view of 
Nelson teaches the method of claim 18. 

Ludwig in view of Nelson does not teach at least one participant can interact with 
the physical device. 

Ayatsuka teaches a video in which a viewed physical device can be controlled 
remotely by a participant (col. 8 II. 23-28). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to incorporate 
Ayatsuka's teaching of using remote camera control and a live video image of a 
device in order to remotely manipulate/interact with devices that are not 
physically reachable. 

Regarding claim 20, when read in light of claim 18, Ludwig further teaches 
each of the at least one participants can interact with the shared interactive 
environment through different computing devices (col. 6 II. 28-51). 

Regarding claim 22, when read in light of claim 18, Ayatsuka further 
teaches a physical device can include a display (fig. 17), a projector (fig. 15). In 
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regards to limitation "a pliysical device can include a printer, a facsimile machine, 
a personal digital assistant, a computer, and a portable computer", Ayatsuka 
further discloses in 10 II. 33-38 that a computer may link to and control any object 
it sees with a network address (col. 7 II. 25). It is well known in the art that 
printers, facsimile machines, personal digital assistants, computers, and portable 
computers can have IP addresses associated with them. 

Regarding claim 23, when read in light of claim 18, Ayatsuka further 
teaches the physical device has associated with it a pop-up control panel through 
which a user can configure and control it (col. 10 II. 49-64). 

1 1 . Claims 24, 26-27, 30, 32, and 34-35 are rejected under 35 U.S.C. 1 03(a) as 
being unpatentable over Emens et al (US 6,463,343 B1) in view of Ayatsuka et al (US 
7,188,139 B1), and further in view of Burt et al. "Object tracking with a moving camera", 
IEEE An Application of Dynamic Motion Analysis. 1989 pp. 2-12, and Nelson et al (US 
2004/0236830).. 

Regarding claim 24, Emens teaches a shared interactive environment, 
comprising: 

a camera system to provide a first live view and second live view (col. 3 II. 8-9, 
where the first live view and second live view are the same in this instance; col. 5 
II. 15-20 teaches multiple cameras for providing multiple different views, or a 
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moving camera for providing different views from one camera); 
a first graplnical user interface (GUI) coupled to the camera system and to 
present the first live view and the second live view, wherein the views can 
capture a physical device (abstract); 

a device controller to dynamically control the physical device in response to 
interaction of a first user with the GUI. 

Emens does not teach the second live view can be configured to zoom in 
on a portion of the first live view; 

wherein the interaction can including annotating at least one of 1) the first live 
view; and 2) the second live view (fig. 2e, col. 3 II. 10-1 1 ); 
wherein annotations are automatically transferred to the physical device in the 
live views if the annotation is at least partially drawn over the physical device as it 

appears in the live video; 

a device tracker coupled to the camera system and to dynamically recognize new 
physical devices; and 

wherein the camera system can be mounted on a mobile, robotic platform. 

In regards to limitation "wherein the second live view can be configured to 
zoom in on a portion of the first live view", official notice is taken that zooming is 
a well known and widely used tool in video surveillance and conferencing 
systems. It would have been obvious to one of ordinary skill in the art at the time 
the invention was made to include panning and zooming capabilities in the 
interactive environment taught by Emens for the benefit of allowing 
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videoconference participants to have access to view of any resources in the 
collective rooms, such as whiteboards or illustrative models. 

Ayatsuka teaches a camera connected to a computer that can detect and 
recognize devices in the camera's view, and allows the viewed device to be 
controlled by the computer (abstract). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to incorporate the 
device detection capabilities taught by Ayatsuka in the device control system 
taught by Emens for the benefit of allowing a user to register devices for control 
by a computer system without manually entering information for each device. 

Burt teaches a robotic camera that detects and tracks a target (abstract). 
It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to incorporate Burt's teaching of a robotic camera that keeps 
targeted objects in its line of view in the interactive environment taught by Emens 
in view of Ayatsuka for the benefit of maintaining a view of devices that the user 
may want to exercise control over. Burt's abstract teaches the application of this 
concept to automated surveillance in the abstract. 

Regarding limitations "wherein the interaction can including annotating at 
least one of 1 ) the first live view; and 2) the second live view (fig. 2e, col. 3 II. 1 0- 
1 1 ); and wherein annotations are automatically transferred to the physical device 
in the live views if the annotation is at least partially drawn over the physical 
device as it appears in the live video" Nelson teaches participants of a 
conference may use annotation software installed on an electronic device to 
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annotate a live view of a physical device (figure 8 display screen 216). The 
annotation will appear to other conference participants and will be projected onto 
display 216 (pg. 8 sect. 0067 and pg.9 sect 0068). It would have been obvious to 
one of ordinary skill in the art at the time the invention was made to include the 
remote annotation capabilities disclosed in Nelson in the videoconferencing 
system taught by Ludwig for the benefit of allowing all participants of a 
conference to input and share all of the resources used to communicate 
information in the conference (such as a whiteboard). 

Regarding claim 26, when read in light of claim 24, Emens further teaches 
the device controller can control the physical device through at least one of: 1 ) an 
infrared communication channel; and 2) one or more networks (fig. 1 col. 5 II. 26- 
31). 

Regarding claim 27, when read in light of claim 24, Emens in view of 
Ayatsuka and Burt further teaches the device tracker can recognize new physical 
devices by at least one of: 1 ) image pattern recognition (Burt pg. 3 col. 2 
paragraph 2, fig. 2); 2) radio frequency transmission; and 3) acoustic signal. 

Regarding claim 30, when read in light of claim 24, Emens further teaches 
the GUI is implemented as one or more web pages (col. 2 II. 60-65). 
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Regarding claim 32, wlnen read in light of claim 24, Emens further teaches 
the physical device has a pop-up control panel that can be made apparent to the 
first user through the first GUI and wherein the pop-up control panel allows the 
first user to control and configure the physical device (fig. 2e). 

Regarding claim 34, when read in light of claim 24, both Emens and 
Ayatsuka teach software based control schemes for controlling objects and 
devices. It is inherent that the physical device can be represented by a set of 
attributes and a set of behaviors, as those are the only ways of representing an 
object to a computer program. 

Regarding claim 35, when read in light of claim 34, Emens in view of 
Ayatsuka and Burt teach the shared interactive environment of claim 34. 

Emens in view of Ayatsuka and Burt does not teach the representation of 
the physical device is part of a device hierarchy. 

Both Emens and Ayatsuka teach software based control schemes for 
controlling objects and devices. Object oriented programming is an obvious 
approach to developing such software, as each device has its own attributes and 
control routines. The concept of classes and inheritance are well known and 
often used to create programs that support a variety of objects. Official notice is 
taken that it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to use objects and inheritance to implement device 



Application/Control Number: 10/629,403 Page 25 

Art Unit: 2423 

control software for the benefit of optimizing the software and reducing 
redundancy in the application programming for devices that may share similar 
functions. 

12. Claim 25 is rejected under 35 U.S.C. 103(a) as being unpatentable over Emens 
et al (US 6,463,343 B1) in view of Ayatsuka et al (US 7,188,139 B1) and Burt et al. 
"Object tracking with a moving camera", IEEE An Application of Dynamic Motion 
Analysis. 1989 pp. 2-12, and Nelson et al (US 2004/0236830)., and further in view of 
Andersson (US 2002/01 1 1999 Al ). 

Regarding claim 25, when read in light of claim 24, Emens in view of 
Ayatsuka, Burt, and Nelson teaches the interactive environment of claim 24 and 
suggests implementing device control in a video teleconferencing environment 
(fig. 5), but does not disclose multiple users having access to the system (i.e. the 
first GUI allows the first user to interact the physical device; and wherein the 
interaction of the first user is apparent to a second user via a second GUI). 

Andersson teaches a system that allows networked computers to access 
and control devices connected to any computer on that network, where both host 
and remote computers can monitor device events (pg. 2 sect. 0019, 0023). It 
would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the remote access and device monitoring 
capabilities taught by Andersson in the shared interactive environment taught by 



Application/Control Number: 10/629,403 Page 26 

Art Unit: 2423 

Emens in view of Ayatsuka for the benefit of enhancing collaboration and making 
a more realistic environment for users who may share resources. 

13. Claim 28 is rejected under 35 U.S.C. 103(a) as being unpatentable over Emens 
et al (US 6,463,343 B1 ) in view of Ayatsuka et al (US 7,1 88,1 39 B1 ) and Burt et al. 
"Object tracking with a moving camera", IEEE An Application of Dynamic Motion 
Analysis. 1989 pp. 2-12, and Nelson et al (US 2004/0236830)., and further in view of 
Hildebrandt (US 2004/0070616). 

Regarding claim 28, when read in light of claim 24, Emens in view of 
Ayatsuka, Burt, and Nelson teaches the interactive environment of claim 24, 
wherein the physical device can be a display (Ayatsuka fig. 5, col. 22 II. 63-67). 
Emens in view of Ayatsuka and Burt does not teach the display can include an 
image stack. 

Hildebrandt teaches an archive memory for use with an electronic 
whiteboard in which images associated with the whiteboard are grouped and 
saved. These images are stored as a stack, where the oldest image is deleted 
when a new image is added (see fig. 22 el 524). It would have been obvious to 
one of ordinary skill in the art at the time the invention was made to store a 
plurality of images captured from a display for the benefit of allowing a user to 
continually use a display resource while still being able to access data previously 



Application/Control Number: 10/629,403 
Art Unit: 2423 



Page 27 



represented on that display. 

14. Claim 29 is rejected under 35 U.S.C. 103(a) as being unpatentable over Emens 
et a! (US 6,463,343 B1) In view of Ayatsuka et al (US 7,188,139 B1) and Burt at al. 
"Object tracking with a moving camera", IEEE An Application of Dynamic Motion 
Analysis. 1989 pp. 2-12, and Nelson et al (US 2004/0236830)., and Andersson (US 
2002/01 1 1 999 Al ), and further in view of Westfield (US 6,677,979 81 ). 

Regarding claim 29, when read in light of claim 25, Emens in view of 
Ayatsuka, Burt, Nelson, and Andersson teaches the shared interactive 
environment of claim 25. 

Emens in view of Ayatsuka, Burt, and Andersson does not teach the first 
GUI can provide a second live view that is different from the second live view 
provided by the second GUI. 

Ayatsuka suggests one environment in which devices remotely controlled 
through a camera view could be used is in a teleconference (fig. 5). 

Westfield teaches a teleconference environment In which a participant can 
see 2 views, where the second view can be specified for each participant 
(abstract, fig 8b, fig 7 and col. 2 II. 46-56). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to include the multiple 
views of an environment taught by Westfield in the interactive environment taught 
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by Emens in view of Ayatsuka, Burt, and Andersson for the benefit of more 
closely mimicking human vision by providing both wide angle and focused views 
(col. 1 11.43-54). 

15. Claim 31 is rejected under 35 U.S.C. 103(a) as being unpatentable over Emens 
et al (US 6,463,343 B1) in view of Ayatsuka et al (US 7,188,139 81) and Burt et al. 
"Object tracking with a moving camera", IEEE An Application of Dynamic Motion 
Analysis. 1989 pp. 2-12, and Nelson et al (US 2004/0236830), and further in view of 
Westfield (US 6,677,979 B1). 

Regarding claim 31 , when read in light of claim 24, Emens in view of 
Ayatsuka Burt, and Nelson teaches the shared interactive environment of claim 
24. 

Emens in view of Ayatsuka and Burt does not teach the first user can 
select the second live view by drawing a diagonal in the first live view. 
Westfield teaches a teleconference environment in which a participant can see 2 
views, where the second view can be specified by a participant dragging a 
rectangle across the image in the first view, (abstract, fig 8b, fig 7 and col. 6 II. 
38-40). It would have been obvious to one of ordinary skill in the art at the time 
the invention was made to include the zoomed view of an environment taught by 
Westfield in the interactive environment taught by Emens in view of Ayatsuka, 
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Burt, and Andersson for the benefit of more closely mimicking human vision by 
providing both wide angle and focused views (col. 1 II. 43-54). 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to JASMINE STOKELY-COLLINS whose telephone 
number is (571 ) 270-3459. The examiner can normally be reached on M-F 9:30-5:30 
EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Andrew Koenig can be reached on (571) 272-7296. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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